Adaptive Prior-Dependent Correction Enhanced Reinforcement Learning for Natural Language Generation

Natural language generation (NLG) is an important task with various applications like neural machine translation (NMT) and image captioning. Since deep-learning-based methods have issues of exposure bias loss inconsistency, reinforcement learning (RL) widely adopted in NLG tasks recently. But most RL-based ignore the deviation ignorance issue, which means model fails to understand extent token-level well. It leads semantic incorrectness hampers agent perform To address we propose a technique called adaptive prior-dependent correction (APDC) enhance RL. leverages distribution generated by computing distances between ground truth all other words correct agent's stochastic policy. Additionally, some techniques on RL are explored coordinate APDC, requires reward estimation at every time step. We find that special case RL, where state transition deterministic afterstate value equals Q-value utilize such prior knowledge, estimate advantage function difference Q-values can be estimated Monte Carlo rollouts. Experiments show that, three (NMT, captioning, abstractive text summarization), our method consistently outperforms state-of-the-art approaches different frequently-used metrics.